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ABSTRACT 

Inappropriate uses in continuing education of Scriven's ^ 
(1967X concepts of formative and summative evaluation are. des* 
cribed and guidelines are presented/ for when formative and - 
summative evaluations are and are not appropriate in continuing 
educatiotjrni&rograms • A role of evaluation analagous to formative 
and summative evaluation but pertaining to programs that have 
been implemented for some time and are now up for review, dubbed ^ 
confirmative evaluation, is described and explained. A non-valid 
evaluation role, justificative evaluation, is also described and 
explained, and guidelines for obViating a need for that role are 
suggested. 



EKLC 



To comprehend the nature and origin of current abuses of 
evaluation in continuing education it is helpful to review the 
phenomenal progress of that newest growth industry in education, 
be it continuing* or pirepp/atory—educational evalua^ion. A decade 
ago or so, in most circles the word evaluation meant the systematic - ^ 
gathering of evidence that a student had learned (or failed to learn) 

c 

-what he or she was expected to learn. In other words ^ evaluation 
dealt with pupil achievement. 

During the middle and late sixties, some rethinking of the 
role and scope of evaluation was done by some noteworthy scholars 
(e.g., Cronbach, 1964; Scriven, 1967; Stake, 1967) and the focus 
of evaluation was subsequently expanded to include the instructional 
materials, curriculum, teaching strategies, and other variables in 
a teaching/learning system. Cronbach's (1964) suggestion that Jthe 
components of a developing curriculum become a focus for evaluation 
data, and Scriven's (1967) expansion of that thought into the 
distijiction between formative and summative evaluation ^represented 
a quantum leap in evaluation ^thinking. As aspects of the teaching/ 
learning system other than student achievement were proposed as 
i'being eligible for evaluation, it became commonplace to speak 
of the new, almost separite , fields of program evaluation and 
product evaluation. ^ 

■■ At f irst , seniority played as large a part in the evolution 

of educational evaluation as anywhere else, and among the first 
(and most-widely re)Spected) of the program evaluators were those 
who had a firm grojindinjg_-iii the- evaluation-of student achievement . 



■ 4 



student achievements being one of the outcomes of a program^ must 
indeed reflect the quality of that program (th3 argument went) 
and must perfort:e be included in amy evaluation of the program; 
evaluators of student achievement saw the ^aovement as simply a 
widening of their roles. BeVlflfe:e# who better qualified to collect 
quantitative data and analyze the^ than those who had already been 
doing it for somewhat different reasons? 

*^ 

The broadening is£ the focus of edu<-;ational evaluation # 
however, had. the effect of proliferating evaluation models # amon^ 
them those that eschewed quantitative methods in favor of othVrs 
(e.g., Eisner^lr 1972, 1975, Note I; McCutcheon, 1976; Rippey, 1973; 
Schwille & Porter, 1976; V?illance, 1976; Wolf, Potter & Baxter, 
1976).^ Indeed the yardstick used for measurement, it was sug-* 
gested, might soitretimec be ignored deliberately {Scriven, 1973 
Stake, Note 2). The qreation of these new models has changed the 
complexion of educational evaluation consir^erably in the last 
decade, but one of the few things that has remained rel^atively 
constant while the field has been evolving is the raison d'etre 
of evaluation: most ey»^luators agree that the function of ev- 
aluation is to pexmit informed decisions-making. 

The movement of educational evaluation gathered momentum, 

. ' . ,■ ■ ' / ^ 

and, figured so prominently in the then current hot issue of 

accountability in decision-making, that agencies other than . 

educational ones looked to the models developed in educational 

evaluation forsalvation with respect to their own concerns .about 



accountability. Consequently # social action programs of various 
types adopted much of the terminology and the orientation of 
education evfiluation, often^ apparently^ without carefully ex- 
amining the consequences of the shift of focus from educational 
curricula to social action programs. Models developed by Stake 
(1967) and Stuff lebeam (1971) were som^^cimes ^dopted wholesale. 
Sometimes the fit ,was relativcaiy good; at other times imaginations 
had to be stretched to adapt the reality to the model. As products 
and programs to be evaluated grew more divergent in nature from 
the origiryl basis on which most evaluation theory was originally ' 
f orged--iSchool curricula^^the sense of th-e ^conciapts and terms 
used in various evaluation model's was distorted by greater and 
greater amounts. One would hear of attempts to do 'summative" 
evaluations of unique (in the sense that they would not happen 
*&gain) events. 

Whether deliberate or not^ this distortion has unfortunate 
consequexH:es for communication within the field. For evaluators 

A 

and program planners to be able to accurately discuss evaluation 
in their projects requires a common terminology. If you will 
excuse a mashed metaphor # a rose by any other name may smell as 

sweet, but the term " formative evaluation • implies certain atr 

■J 

tendant conditions which have not always beun met in recent 

... •» 

practice. 
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Pr<»requi3ites for Formative Evalxiation 

By dexlnltlon, formative evaluation Is evaluation whose role 
Is to ".. •discover deflcl^cles and successes In the Intermedia te 
versions of a new [product] ... (Scrlven^ 1967 r p. 51)"; formative 
evaluation is a par?: of the process of devaiopment of ,a new 
products be it a curriculum^ a textbookr a television series^ or 
a seminar. I'want to emphasize two things about the definition: 
the first is that it takes place during the development of the 
product; the second is that it- takes place with respect to a 
prcduct. These two points sometimes* get overlooked when we speak 
of formative evaluation in continuing education contexts. 

Let • s .consider the first point — that formative evaluation 
takes place during, the development of the product.. The systematic 
development process (e.g.r see Baker^ 1974r 1977; Baker is Schutz^ 
1971; Markler 1967) requires that an instructional" prodAict be 
conceived and largely developed ixi isolation \from the target learner : 
a firi^t draft is devised^ making i^it as good as possible by taking 
it through as manytevisions as are— fudged necessary to bring it 
to the prototype siage; the prototype is then given, under careful 
supervision, to one or two learners (who are not memii^rs of the 
target audience themselves , "but who are similar to those members) 
to subject it to a preliminary field test. Ail qf this happens 
before the design is implemented with the learners themselves — 
to dp otherwise would raise serious moral and ethical concerns. 



(To .be sure# it may be— *ought to be— *field tested later with a 
larger number o£ learners ^ probably sampled £rom the target 
groupr but by this time the design ought to be fairly stable; 
few modifications to^~the basic^e^sign are to be expected at this 
stage.) 

Now let^s consider the second point. Note that formative 
evaluation is meaningful only when the object of evaluation is a 
product — a durable, replicable entity. Typically, we think of 
a product as a tangible obj ect (e.g., a book, a film, a handout, 
etc.). However, it is also pL>ssibl^ to conceive of a sequence of 
activities or a. series of steps (i.e., what we frequently call a 
process) 4S a product, so long as the sequence of activities or - 
series of steps can be repeated time and again with only minor,^ 
if any, variations. This interpretation, which will be used 
throughout this paper, permits the consideration of a j>erson*s 
activities as a product, so long as the condition of relative 
invariance is met. 

The product, whether tangible or not , musto have , abovt^ all, 
predictable and replicable^ef f ^cts ti»e., it must be consistent 
in its operation) if we ar6 to apply formative evaluation to it. 
Sisiply put, formative evaluation can be conducted on an ed- 
ucational product if and' only i£ the' product, can be applied 
time and txme again in substantially the same form and manner 
(i.e., it is durable) with substantially the same result (i.e.. 



its effects . are repllcable] • This means that it is not, meaningful 
to speak of formative evaluation with respect to an activity that 
is interactive or dependent (e.g. # as is^Jtlxe- case when instruction 
tak^s different directions ^response to certain events, such 
as expressions of .learner Interest) • I am not saying that teach-* 
ing should^ never be interactive? I am saying that if it is, it is 
meaningless to speak of its formative evaluation. 



Summative and^onf irmative Evaluations 

Summative evaluation , ^n the other hand, is the evaluation 
of a finished product (Scriven, 1967, p^ 40-43) . The product, by 
the time summative evaluation is performed, has been developed 
and field tested, and is in more or less immutable form. The 
basic decision to he made onVthe basis of a summative evaluation 
.is "Should we implement this product (or program, seminar series, 
etc«)?'* A summative evaluation demonstrates for public consump-^ 
tioa what the product is capable of doing • The ^information will 
not be used by the product developrers^ to make modifications to ^the 
product; it will be used by administrator s to decide whether or 
not they will adopt the product and. put it i^nto use. " 

7h«re is another evaluation role, one extant in both'pre^.. 

• : 'i ' 

para'tory education and in continuing education but perhaps more 
evident in the latter, that Scriven*s distinction does not cover--* 
the situation in^hich a product (which has undergone formative. 



and possibly summative, evaluation) has been implemented and op- 
erated for some .time, and the time " has, come to review its efficacy. 
;,The evaluation question to be asked is "Should this program be 
discontinuedr and, if so, should it be replaced?" the negative 
answer to the first part of the evaluation question begs^ the 
second question: "If it should be continued, should it be con- 
tinued in its present form, or should it be modified (and if so, 
how)?" This role of evaluation, which I propose to call coii- 
tiralative evaluation, is substantially different from either 
formative or summatiye evaluation. There is> first, the aspect 
to confirmative evaluation 6tf final judgment (accept it as" is, 
or . reject it) that, is implicit' in the qjues^tion "Should the program 
be discontinued?", as there is in a summative evalu'at.ion. But 

tliere is axso an aspect of formative evaluation implicit in a 

♦ .- 

confirmative evaluation if the answer to the first question is 
negal:ive1» since the secqnd questicr^ -asks hoW the program might 

^ • 

be revised, if indeed revision is indicated. This contingency 

■ \ ^ ' . . 

of evaluation questions makes the role of confirmative evaluation 

'. . - 

f®^®nt from the roles of either formative or sximmative evaluation 
4 . ■■■ ■ • ^ 

To call a confiraative evaluation a supimative evaluation 

followed by* -a formative evaluation seems inconsistent wilth tjie 

original .aeaniAg of the words: by dofdnition, formative evaluation 

takes ^lace before the completion of ^ product, while summative 

■ \ 

evaluation takes place! after_compIetion. Ergo, a formative 



evaluation of a product cannot succeed its summative evaluation.' 

« c c ^ * 

Although it is £>ossible to conceive of a. situation in v«hich 
summative , evaluation is done wit-hout any; formative evaluation 
having, preceeded it, such a scate of affairs seems quite xncon- 
sistent withr-th'e situation Scriven described* Similarly # it / . 
would De possible to conceive of a confirmative evaluation being 
done on a prp'grifia that had the benefit of neither formative nor 
summative. evaluation. In such a situation^ it might be tempting 
to regard the confirmative evalua^tion as formative. However # 
in the reality of program dev'elopment ^ formative evaluation is 
seldom, if ever, 'conducted with the . intention (or even possibility) 
of discontinuing the development^ of the. program in response to 
evaluation findings. In other words, the program development 
process is such that, if properly Executed (i.e., if a proper 
needs id^entif ideation is done initially) , there is v^ry little 
likelihoqd that formative evaluation information would lead to 
the discontinuance of program development. Rather, formative 
evaluation findings would lead to modification of the program 
^being developed. Inxconf irmative evaluation, of course, a very 
real possibility exists thaj: ^he data -would indicate ^hat the\ ' 
^rogr^m should be discontinued, witliout replacement . . Hence ' 
the distinction among formative, summative, and confirmative 
eyaluatxons seems necessary .c ' u . > 

' Itt addition, there are^ragmatic concerns that help to , . 



differentiate confirmative from formative and summatrve evaluation • 
Who should: perform confirmative evaluations-^someone intimatefy 
involved vith the prograto Cas- in formative evaluation! pr someone 
quite divorced from it Cas in summative evaluation) ? It would 
seem that the relatively, high degree of objectivity presumed 

of the summative evaluation ought to be applied to the conf ir- 

• • - - 

mative evaluation decision regarding whether thfe on**going program 
ought ^to be terminated or continued/ but the formative evaluator • 

" ■ ' • ■ r - . 

would be in a much better position tp make the judgment whether 

' ' -. / ^ . ■ . . . ^. 

the program ought to continue intact or be mo.dified (and how^ if ' 

' . *" .. 

it is) , given that the first part of the evaluation question 
indicated 'colStiAuance. * : : ^ * 

At ^he present time^ there are no definitive answers to the 
question.^ Hopefully ^ further experience with confirmative 
evaluations will provide the needed guidelines. 

Justificative Evaluation / 

Finally, I have encountered on-numerous occasions a 'demand for 
an evaluation role whiqh I believe pught not to be filled at all. ^ 

In continuing! educatioi^ programs, it is not uncommon to • \ '\- 

^nd a type of •evaluation in which questions such as the foriqwing 

' ^ . -'^ . ■ ■ * \ • ^ ~ 

are asked:' "How jjrelevajit did you find the section on> . ";Kbw 

" ■ . * * ■ * ■ ' ' 

useful will the information learned in this workshop be in your 
present job role?" ^ or, Bven moris blatantly ^ •'Dp you feel you 



learned enough during this seminar to justify yout attendance?" 
Aside from pratically begging the question, such inquiries, 
typically made at the conclusion of the learning experience, act- 
ujilly ask for informatioh that shouJLd have been fathered prior to 

dubL^ad such "evaluation", 
attempts jastificative evaluation, because they seem most often 
to be a^cempting to justify the design of the event aftei: the 
fact. 

Despite the name I am suggesting, I do not believe that 
justificative evaluation should be considered a valid evaluation 
role at all. The .identification of needs is an; important part 
of the planning process, bi2t the need identification process 

ought to happen before, not after, the design an^ implementation 

■J - 

of the event. ' 

. '"* " ■ . * » , 

Purtheifmare, it is Usually possiijle ta get some positive 
response about an educational experience at its conclusion (if 
nothing else, participants are rather glad it/s over!), but it 
seems to stretch credibility that this data could be considered 
part of objective needs identification. Most people typically 
confusrev. pleasant experiences with valuable oneti and are there-*^ 
fore lik^y to' judge the woJPth of the leatning in terms, of its 
enjoyment.^^ * ^ 

Returning to the stated purpose of evaluation — to make ' 
decisiQha— -and examining justificative evaluation in the light 
of that purpose; I can only conclt de that justificative 



evaluations are purposeless. Any decisions about the design of the 
experience that come out of the proc^iss are bound to be useless: 
with respect to the ekperlence itself, since the experience j 
is already^ over. No knowledge of value vis a vis the needs of 
the learners has been gained that could not have been gainf::d 
more easily and more accurately by other, means (i.e., by doing 
an adequate needs identification). (Those who are tempted to 
argue that something is learned from the process-**.the efficacy 
w£1% which the programmer can "sense** what needs exist-»are 
confusing eveiluation of the program with the ev^.luation of the 

*pro'grammer5> * ability to identify needs mystically.} 

. ' " ' ■ <> 

The close relationship between the identification of 

■ ' . I- • 

' educational needs and the process of evaluation has been noted 
CKaufman# 1977). HoweVer, the two processes are not identities; 
rather, they appear to be two sides of the same coin. They share 
some import2int similarities-- their general raison d'etre of 
providing information for decision-making , and their requirements 
for objectivity, validity, reliability, and utility — ^but'they 
also have' an important difference: -needs identification ought: 
to take place prip^ to the planning phase, whereas evaluation 
ought to take place during the development phase .(formative 
evaluation). Immediately after the development^ phase and "hefore 
the implementation phase (summative evaluation) ; or during the 
implementation phase (confirmative evaluation) . Thus-^L.^^the need 
for justificative evaluation is obviated. 

ERIC 



Should Evaluatio n Be Done? 
— — ^ '. — 

Weiss (1972) identifies several types of uses for evaluation 
couched in terms of particular decisions to be made, that are well 
worth reiterating: 

1« To continue or discontinue the program 

2. To improve its practices and procedures ^ 

^ 3« To add or drop specific program: strategies and 
techniques 

4. To institute similar programs elsewhere 

5. To allocate ^^resources among competing programs 

6. To ?;ccept or reject a p;rogram, approach or theory 
(p, 16-17) ' ' 

' ' s 

Couched in terms of the three role descriptors discussed 
earlier, Weiss' list, . essentially a list of implicit questions 
which constitute reasons for doing evaluations, can be classified 
as formative (2, 3, 6):, summative^ (4, '^nd possibly 5 depending 
on the context), or confirmative (1, f)ossibly 2' if it immediately 
follows 1 in an on-going program, and possibly 5, depending on 
the context) evaluation questions* The list suggests reasons why 
an evaluation might be done, but does not suggest any cbnditiohs 
under whieh it might be i;easonable bo consider doing the eval- 
uation* Such a list of conditions would be useful. In the past« 
ftttlf-dozen years, evaluation has become a buzz-word in public 
agencies of various kinds. For some . reason (lack o£ clarity in 

15 



13. 

communication?, lack of understanding of the concepts?) eval*- 
uation requirements are often attached willy-nilly to projects # 
with seemingly little concern for their usefulness or meaning. 
Projects that should not or cannot be evaluated xSeaningf ully 

• , ' • ' ■ ! . 

are bein^ required to have evaluation/components. Perhaps the 
following list of sugg;^steGt. conditio/hal rules will help to 
ameliorate the problem: / 

- ■•■ .■ ■ ■ - /.■■ ■ 

1« Formative evaluation should be performed only if the 

•-. - • / "■ ■ 

ol^ject of evaluation is a durable, replicable entity . 



r 



If the entity is not capable of being presented time 
and again in largely the same form and manner with 
largely the same effect, formative evaluation is 
probably meaningless and a ^waste of V.ime and mbxiey. 
Although durable, replicable entities are often ' 
referred to as products , recognize that human beings 
and the processeis they engage in can we^l be part of 
thie package, provided that their actions .are replicable 
and that their involvement in the activity is durable. 
2. Summative evaluation should also be perforated only if ' 
the object of evaluation is a durable, replicable 
entity that has been sub j ected to formative evaluation 
. earlier. Summative evaluation is designed to demonstrate 
what, a product is citpable of doing in extant fbrm. 
Obviously if the form is malleable (i.e., the product is 
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not durable and/or replic^ble) summative evalue^tion 

is a waste of time since tIJfere is no reason to believe 

that subsequent or previous applications replicate the 
■ .. . . t 

application under evaluation. rurthermorer to, perform 

a summative evaluation an a (durable, replicable) product 

that has not been subj octed to formative evaluation is 

o- 

relatively cos c-inef f ective\ Negative results from such 
a summative evaluation could be attributed to sub- 
optimal design thna't could perhaps be remedied easily 
and inexpensively if the flaw were determined through 
formative evaluation. v . 

Conf^irmative evaluation may be performjad ,qn a product 
or program that has not had the benefit of either 
lormative or summative evaluation (although the fact 
:nat iv" Iv^ £)0?!i?ible should not be taken to mean that 
it A tSftffirHnii?; to imjplement i:^evaluated products or 
prc^:;aifis axx • :;At>.r perform confirmative evaluation 
on them) • But conifirmative evaluation sho'uld be 
^undertaken only on a program that has been in plac^ 
for some time and whoAe cutrent effi^cacy is validly 
in dbubt. Furthermore r the program should have the 
capability of operating unchanged for another period 
of time. Obviously, the periods of time and validity 
of. doubt can only be judged in relation to such variables 



as costf Importance of the progifam content, number 



of people Involved, etc. 
4. Justificative evaluation should not be per^brmed 
. under most circumstances. Prior identification and 
analysis of learning needs ,:l8 far more^i p^l^ter^ble and \ 
ethically responsible. ' y 
Implicit in the above set of rul^SL-are the following, 
alioiost heretical notions: .. . 

1. Neither formative ^valuation noi. summative evaluation 
pught to be attempted for a uni4^e ("one-shot") event. 
Since a "one-shot" event cannot be subjected to 
confirmative evaluation, and justificative evaluation 
sjhould never be undertaken, it follows that "one-shot" 
\ events ought not to be evaluated at all. 

2 . Formative "evaluation should not be und^^ unl'ass 
there is a high probability 4h*t :jthe- object of 
evaluation (event, j^rpduct/ 4^'^ repeated 
' in the iminediate future. i^ Wit^^ real prospect 

of durability and replicabii^ty of effects, formative 
evaluation is a waste of timfi and money. , 
3. Swniaativ^ evaruation should be^'i^ oply: when 

you are considering adopting a ^^^P 
_ , alsewhere, c^^^^^ a product 

that you ha^e develbi»ed. In the latter .case, the 
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evaluator should be external to the developing agency 
or group (i.e.# you should not'^do the sumxaativV^' " 
''gyaluation of the product you developed) • 
If the suggestions* in the paragraph above Were taken 
seriQud>ly and applied widely in continuing educatii^d|i, we woald 

probably have far fewer evaluations taking place# but the . 

' ' ' >' ■ • 

proportion of useful and meaningful evaluations would probably 

rise dramatically ^ 
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